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ABSTRACT 


A  research  experiment  was  conducted  to  investigate  how 
accurate  a  voice  recognition  system  must  be  for  daily 
production  use.  Specifically,  the  purpose  of  the  research 
was  to  establish  the  percentage  accuracy  level  at  which  a 
user  becomes  frustrated  and  decides  not  to  use  a  voice 
reccgnition  device.  The  experiment  consisted  of  controlling 
the  perceived  recognition  accuracy  of  a  voice  recognition 
system  and  then  collecting  data  through  the  use  of  a 
guestionnaire  from  the  experimental  users  on  thc 
acceptability  of  the  equipment.  The  experiment  was  not 
totally  successful  for  a  variety  of  reasons.  This  paper 
will  discuss  the  research  methodology,  review  the  data 
collected,  and  suggest  possible  alternatives  to  the 
experimental  design  to  overcome  the  problem  areas 
encountered. 


User    Accuracy    Requirements 
For    Voice    Recognition 


I.     INTRODUCTION 

Within  the  last  ten  years  voice  recognition  technology 
has  grown  from  a  laboratory  research  endeavor  to  a  useful 
and  economic,  computer  human  interface  tool.  The  equipment 
available  today  is  relatively  inexpensive,  compact,  and 
accurate  as  evidenced  by  numerous  applications,  both 
industrial  and  military,  which  successfully  use  this  input 
methodology.  Improvements  in  the  technology  are  still  being 
made.  Three  major  areas  for  continued  research  in  voice 
input   to    computers    are   as    follows. 

1.  User    independence 

2.  Continuous   speech 

3.  Vocabulary    capability    of    10,003    to    12,000    words 

The  literature  indicates  that  the  SDlu+ion  or  at  least 
incremental  breakthroughs  to  these  three  research  areas  are 
just  a  few  years  away.  Hopefully,  the  goal  of  manufacturers 
and  researchers  is  to  develop  and  produce  systems  with  all 
or  some  of  the  properties  listed  above  which  have  the  best 
accuracy  rate  possible.  A  question  arises  with  this  goal  in 
mind.  At    what      accuracy   rate      should   the      system    be      made 

available      to      users?  If      an      accuracy      rate      of      95%      is 

acceptable  why  not  use  those  systems  while  new  and  better 
algorithms  are  being  developed?  Conversly,  if  95%  accuracy 
rate  is  not  acceptable  and  will  give  voice  recognition  a  bad 
reputation  among  management  personnel  and  users  it  should  be 
held     back    until      an    acceptable      rate   can       be   achieved.  A 

measure      of    where      this   accuracy      threshold    is      could    be      of 


great  use  to  researchers  and  system  manufacturers. 
Therefore,  the  question  investigated  was  how  poor  must  a 
voice  recognition  system  be  before  the  user  becomes 
frustrated  with  the  error  rate  and  will  choose  not  to  use 
the  system. 

A  required  accuracy  rate  will  depend  on  the  task  for 
which  the  system  is  intended  to  be  used.  For  example,  if 
voice  recognition  is  to  be  used  for  the  input  of  guidance 
parameters  and  launch  seguence  commands  in  a  computerized 
missile  delivery  system,  anything  less  than  99+%  accuracy 
would  be  unacceptable  even  under  the  high  stress  situations 
most  likely  existing  during  the  input  process.  On  the  other 
hand,  there  are  numerous  tasks  which  cap.  be  labeled  as  non- 
critical  where  voice  recognition  errors  can  be  tolerated  and 
in  fact  now  occur  frequently  using  more  conventional  input 
methods  such  as  a  keyboard.  A  typical  task  in  this  category 
would  be  information  retrieval  such  as  a  stock  broker 
obtaining  stock  information  for  a  client,  or  an  airline 
reservationist  retrieving  flight  information  for  a  customer. 
Neither  of  these  tasks  are  particularly  critical  in  nature. 
If  an  occasional  error  is  made  during  the  input  process  it 
is  easily  corrected  and  the  process  repeated  without  any 
damage  to  the  system  or  loss  in  revenue,  profit  or  system 
efficiency. 

An  experiment  was  developed  to  study  the  user 
acceptabiltiy  guestion  with  the  objective  of  determining  a 
user  reguired  accuracy  rate.  A  VET/2  voice  recognition  unit 
manufactured  by  Scott  Instrument  Inc.  of  Denton,  Texas  was 
used  in  conjunction  with  a  Basic  software  program  created  on 
an  Apple  II  Plus  microcomputer.  The  experiment's  subjects 
were  asked  to  read  an  ordered  list  of  words  into  the  voice 
recognition  unit  and  observe  whether  the  word  was  recognized 
properly.    This   same  ordered  list   of  words  was   stored  in 


computer  memory.  Each  subject  was  led  to  believe  the  system 
was  in  a  recognition  mode,  but  in  fact,  the  software  program 
waited  for  a  verbal  input  and  then  draw  a  random  number  to 
determine  if  the  spoken  input  should  be  displayed  as  correct 
recognition,  misrecogni tion  or  nonrecognition.  If  the 
random  number  determined  it  was  to  be  a  correct  recognition 
the  next  word  on  the  ordered  list  was  displayed.  If  a 
misrecognition  was  to  occur  an  alternate  word  was  displayed, 
and  a  guestion  mark  was  displayed  if  the  random  number 
determined  a  nonrecognition  was  to  appear.  In  other  words, 
the  program  can  be  viewed  as  a  voice  actuated  system  where 
any  verbal  response  would  trigger  the  recognizer,  but  a 
random  number  would  determine  the  output,  not  the 
recognition  algorithms.  As  long  as  the  subjects  continued 
to  read  the  ordered  list  of  words,  it  appeared  as  if  word 
reccgnition  was  being  accomplished.  By  varying  the  random 
number  test  the  subject's  perceived  accuracy  rate  could  be 
controlled.  The  errors  were  recorded,  and  each  subject  was 
asked  to  complete  a  guest  ion nair e.  The  questions  were 
designed  to  indicate  when  a  frustration  level  was  reached 
due  to  recognition  inaccuracy. 

The  experiment  did  not  lead  to  the  desired  goals  or 
results.  This  paper  will  cover  the  experiment  and  why  the 
actual  results  were  different  than  predicted.  First,  +he 
report  will  describe  the  computer  software  program  which  was 
created  to  vary  the  perceived  recognition  rate  during  the 
experiment.  Next,  the  experimental  design  will  be 
discussed.  The  entire  design  was  not  implemented  because 
preliminary  data  analysis  indicated  the  desired  results  were 
not  being  obtained.  The  method  used  to  implement  the 
experimental  design  will  also  be  discussed.  The  third 
section  of  the  report  will  cover  the  preliminary  data 
analysis  and   summary  of  user   responses  to   the  experiment. 


Finally,  conclusions  will  be  drawn  as  to  why  +  he  experiment 
did  not  resolve  the  question  at  hand,  and  recommendations 
will  be  presented  for  future  research  concerning  the 
question  of  "How  good  should  voice  recognition  ^quipmant 
be?". 


II.     PROGRAM    SOFTWARE    DESCRIPTION 

Two  computer  software  programs  were  written  for  use  in 
this  experiment.  The  first  prsgram  was  used  to  create  three 
databases  consisting  of  words  to  be  used  for  the  planned 
experimental  design.  Since  this  program  was  used  only  as  an 
aid  for  the  database  preparation,  it  will  not  be  discussed. 
A  copy  of  the  program  and  the  databases  of  words  are 
attached  in  Appendix  A  and  Appendix  B  respectively.  The 
second  program  created  was  used  to  alter  the  perceived 
reccgnition  accuracy  of  the  Scott  Instrument  VET/2 
recognizer.  It    was      written      in      Applesoft    Basic      and      is 

included  as  Appendix  C.  The  following  description  explains 
the  program  logic  using  the  program  line  numbers  for 
reference. 

line  10  dimensions  a  character  array  called  W$ (200,2). 
During  the  experiment,  this  array  held  the  200  vocabulary 
words   used    for    the   experiment.  W$(I,1)    held    the   word    which 

was  to  be  spoken,  and  W$(I,2)  held  a  sound  alike  word.  The 
use  of  this  sound  alike  word  will  be  explained  later.  The 
array  F$(3)  held  the  name  of  the  three  word  databases  which 
were  available    for    use.      These    databases    were   named: 

1.  COMPUTER,  designed  to  be  used  by  users  of  e 
text  editor  on  the  IBM  3033  computer  at  the  Naval 
Postgraduate    School, 

2.  STOCK,    designed    to    be    used    by    stockbrokers,    and 

3.  AIRLINE,  designed  to  b?  used  by  airline 
reservationists   or   travel    agents. 

The  array  P(U)  held  the  four  different  probabilities 
associated  with  the  planned  recognition  rates  of  99%,  95%, 
90%,    and    85%. 


Line  20  sets  the  variable  D$  to  a  control  D  which  is  used 
for  file  manipulation  on  the  Apple  computer. 

Lines  30  and  40  are  data  assignment  statements  for  the  F$ 
and  P  arrays. 

Line  45  assigns  to  the  character  variable  M$  a  series  of 
five  blank  spaces.  M$  is  used  for  print  control  or  print 
spacing. 

Line  50  is  an  output  statement  asking  the  uscr  what 
database  is  to  be  used. 

Line  60  accepts  as  an  input  the  number  of  the  desired 
database.   This  number  is  placed  in  the  variable  A. 

Line  70  opens  the  correct  database  file. 

Line  80  sets  the  read  device  to  the  appropriate  database 
file. 

Line  90  reads  in  the  2  00  word  vocabulary  and  their  sound 
alikes  into  the  W$  array. 

Line  100  closes  the  input  database  file. 

Line  105  asks  for  a  random  number  generator  seed  and 
places  the  integer  response  into  the  variable  called  IS. 

Line  110  asks  for  an  algorithm  number,  which  in  effect  is 
the  array  index  of  the  desir°d  accuracy  rate.  The  question 
in  line  110  was  seated  in  such  a  manner  so  tha+  experiment 
participants  would  assume  different  word  recognition 
algorithms  were  being  tested. 

Line  120  places  the  algorithm  number  in  the  variable  B, 
and  if  B  is  less  than  zero,  the  program  is  stopped. 

Lines  130  through  160  call  the  Scott  instrument  voice 
recognition  subroutines  used  to  load  the  voice  patterns  into 


memory,       and      initialize    the      recognition    unit.  rhe   voice 

patterns,  although  not  used  for  recognition  purposes,  were 
neccessary    for    the    proper    operation    of   the    recognition    unit. 

Line    170    prints    a   header      announcing    the  practice    session 

of    10   words.         During   this  practice    session  the    use    of    voice 

recognition      equipment      was      explained        to  the      experimen4- 

participant.  The  explanation  given  to  each  participant  will 
be    described   in   detail    later. 

Lines       180      through      2  00    create      a      program      loop.  The 

variable  I  is  used  as  an  indexing  variable.  This  indexing 
variable  is  first  set  to  1,  and  the  first  word  of  +he 
vocabulary  database  is  printed  on  the  Apple  computer 
display.  This      display      is      used      .as      a      prompt      to      +he 

experiment's  participants  for  the  word  they  are  to  speak. 
After  the  word  is  displayed  the  program  is  transferred  to 
the  recognizer's  subroutine  which  will  accept  a  verbal 
input.  After  the  recognizer  accepts  the  voice  input  the 
subroutine  returns  control  to  this  main  program.  After  the 
acceptance  of  the  voice  response  an  artificial  delay  is 
created       by    the      "FOR      Z    ..."      statement.  This    delay      was 

neccessary  to  provide  a  capability  of  stopping  program 
execution  if  the  participant  made  an  error  which  could  lead 
them  to  believe  the  program  was  not  actually  recognizing 
their      voice.  The        delay      provided      the      time        for      ^he 

experimenter  to  stop  the  program  before  the  "recognized" 
word    was    displayed.  After    the  delay   a      subroutine,      which 

will  be  described  later,  is  called  to  determine  whether  a 
correct  response,  a  sound  alike  mistake,  a  random  mistake, 
or  a  nonrecognit ion  response  should  be  displayed  to  the 
participant.  After    the      response    is      displayed   the      index 

variable,  I,  is  checked  to  determine  if  the  test  practice 
session    is   over. 


Line      210   sets      a      series   of      counters      to   zero.  These 

counters  keep  track  of  the  number  of  non  recognitions 
presented  to  the  participant  (Q),  the  number  of  sound  alike 
misrecognitions  presented  (S)  ,  and  the  number  of  nonsense 
misrecognition  responses  (N)  .  The  variable  T  is  calculated 
at  the  end  of  each  particpant1 s  pass  through  the  200  word 
vocabulary  list  and  holds  the  accuracy  rate  actually 
presented    to   the    subject. 

Lines  220  through  250  comprise  another  program  loop.  The 
logic  is  similar  to  lines  180  through  2D0  except  the  entire 
200    word    vocabulary    is    seguentially    displayed. 

Lines  260,  270  and  280  respectively  display  statements 
thanking  the  participants,  calculating  the  actual  accuracy 
rate  presented,  and  displaying  the  accuracy  rate  and  all 
counters    in    a    coded    form. 

Line  290  sends  control  of  the  program  back  to  the 
guestion  asking  which  algorithm  should  be  used  (Line  110), 
and    the    program   is    ready    for    the   next    participant. 

Line  300  is  the  first  line  of  the  subroutine  which  will 
calculate  whether  the  response  which  is  to  be  shown  to  the 
participant  is  a  correct  recognition  response  or  one  of  the 
three  possible  error  responses.  A  random  number  is  drawn. 
If  the  random  number  is  greater  than  the  accuracy  rate  which 
is  presently  being  simulated  the  program  will  branch  to  the 
statements  neccessary  to  calculate  the  type  of  error  which 
should    be   presented. 

Line  310  is  executed  if  a  correctly  recognized  response 
is  to  be  displayed.  The  print  statement  will  first  prin*- 
the  variable  containing  the  blank  spaoes,  and  then  the 
correctly  recogniz°d  word.  This  was  done  so  the  recognized 
word   was    displayed    further   to  the      right    on    the    Apple    screen 


than  the  word  which  was  output   as  a  prompt.    Transfer  then 
is  passed  to  statement  400. 

Lines  320  trhough  340  are  used  to  determine  the  type  of 
error  that  should  be  displayed  given  that  the  present  word 
is  to  be  perceived  as  an  error  by  the  participant.  A  random 
number  is  drawn,  and  if  it  is  less  than  .33  the  error  is 
considered  a  sound  alike  error  and  control  is  passed  to  line 
390.  If  the  random  number,  R,  is  between  .33  and  .56,  it  is 
considered  a  random  or  nonsense  error  and  control  is  passed 
to  line  360.  Finally,  if  the  random  number  is  greater  than 
.66,  it  is  considered  a  nonrecognition.  Therefore,  the 
three  types  of  errors  are  equally  likely.  In  previous  voice 
recognition  studies,  Poock (  1980)  ,  Poock(1981),  and 
Jay (1981),  error  rates  of  about  1.8^  have  been  consistently 
experienced  with  a  Threshold  Technology  Inc. ,  Delran  New 
Jersy,  model  600  voice  recognizer.  In  these  studies  non- 
recognitions  consisted  of  3  1%  to  35%  of  the  total  recorded 
errors.  There  were  no  statistics  available  on  the 
percentage  of  misrecognitions  which  could  be  considered 
sound  alikes  or  non-intuitive  confusing  misrecognitions. 
For  this  reason  it  was  assumed  that  misrecognitions  should 
be  egually  divided  between  the  sound  alike  possibilities  and 
the  random  error  possibilities.  Therefore,  all  three  error 
types  were  programmed  to  occur  with  equally  likely 
protabilities. 

Line  3  50  is  executed  if  a  non  recognition  is  +  o  occur. 
It  again  prints  the  variable,  MS,  which  contains  blanks  and 
then  a  question  mark,  ?,  representing  the  Scott  Instrument 
convention  for  a  non-recognition.  The  counter  for  non- 
recognitions  is  increased  by  one  and  the  program  is  then 
transferred  to  line  400. 

Line  360  through  380  determine  a  random  word  response  for 


the  random  misrecognition  case.  A  random  number  is  drawn 
and  converted  into  a  random  integer  between  1  and  200.  N^xt 
it  is  checked  to  ascertain  that  the  random  integer  generated 
is  not  the  word  which  is  to  be  misrecognized.  If  this  check 
had  not  been  made,  a  misrecognition  could  have  been  recorded 
but  the  participant  would  have  in  fact  seen  the  correct 
response.  The  randomly  selected  word  is  printed  on  thp 
display  in  the  same  manner  as  a  properly  recognized  word, 
and  the  nonsense  word  counter  is  incremented.  Again  control 
is  then  passed  to  statement  400. 

Line  390  is  executed  if  the  random  number,  R,  indicates 
that  this  incorrect  recognition  should  be  a  sound  alike  and 
the  second  word  in  the  W$  array  is  printed  out.  These  words 
have  been  selected  in  such  away  that  an  average  user  would 
conclude  that  it  was  an  understandable  or  likely  recognition 
errcr.   The  sound  alike  counter  is  incremented. 

Line  400  is  the  last  line  of  the  subroutine.  Again  a  loop 
is  added  to  produce  a  delay.  The  index  used  in  the  program 
loop  consisting  of  statements  220  through  260  is 
incremented.  A  blank  line  is  displayed  for  readablity  and 
the  subroutine  returns  to  the  main  program  to  print  the  next 
word  in  the  200  word  vocabulary  list. 

The  next  section  will  describe  how  this  program  was  used, 
and  the  reaction  of  the  experiment's  particip?nts  as  to  the 
believability  that  a  voice  recognizer  was  being  tested. 
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III.    EXPERIMENTAL    DESIGN 

It  has  already  been  mentioned  that  ^hree  databases  w=>re 
formed    for      the   research    experiment.  Thp    plan    was      to    rur. 

three  groups  of  100  people  each  through  one  of  the 
databases.  In  other  words,  100  stockbrokers  or  investment 
counselors  would  say  the  209  words  associated  with  their 
profession,  and  then  rate  *-he  acceptability  of  the 
eguipment.  Likewise,  100  airline  reservationists  or  travel 
agents  were  to  use  the  airline  reser vationist  word  list,  and 
100  students  would  use  the  200  words  associated  with  the  IBM 
3033   text    editor    program. 

A  word  list  of  200  words  was  used  for  two  reasons.  First 
it  was  decided  that  going  throuoh  a  list  of  100  words  went 
too  fast,  and  the  subjects  would  not  get  a  good  feeling  for 
the   accuracy   rate.  Using   300    words   was      definitely    out    of 

the  question  because  of  the  time  involved  in  conducting  the 
experiment  for  the  number  of  planned  subjects,  and  because 
of  a  possible  boredom  factor  which  could  complicate  the 
subjects  perception  of  the  system.  The  median  seemed  like  a 
reasonable  choice.  The  vocabulary  size  of  200  words  also 
satisfied  a  second  criteria.  That  was  the  desire  to  get 
user  frustration  information  at  a  more  accurate  level  than 
every  1  percentage  point.  At  least  the  200  words  would  give 
ratings    at   every   one   half    percentage    point. 

During  the  experiment  it  became  apparent  that  a 
frustration  level  was  not  being  achieved  or  at  least 
measured.  Therefore,         after      the      first      fourth      of      "-.he 

experiment  some  preliminary  data  analysis  was  done,  and  the 
results  showed  that  little  was  being  learned  about  user 
accuracy  needs.  It  was  decided  to  stop  the  experiment  and 
report      on   what      had      been  dona      to    date.  A      total    of      78 
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subjects  participated  in  the  test.  The  intent  of  this 
section  of  the  paper  is  to  explain  how  the  subjects  were 
introduced  +o  the  experiment  and  how  they  reacted  to  the 
reccgnition  system.  This  section  will  also  discuss  the 
design  of  the  guestionnaire. 

The  subjects  were  students,  staff  and  faculty  members  at 
The  Naval  Postgraduate  School  in  Monterey,  California.  They 
were  all  volunteers  and  between  the  ages  of  25  and  49.  The 
entire  explanation  and  experiment  took  between  10  and  15 
minutes  per  subject. 

When  a  subject  arrived  at  the  experiment  site,  it  was 
explained  that  some  new  user  independent  voice  recognition 
algorithms  were  being  tested.  The  Scott  Instrument  voice 
recognition  device  was  covered  to  preclude  the  subjects  from 
getting  the  wrong  impression  of  its  capability.  The 
algorithm  number  was  entered  into  the  system  in  front  of  the 
subject  as  the  idea  of  different  algorithms  was  being 
explained . 

Some  of  the  subjects  had  used  voice  recognition  eguipmen*-. 
on  previous  voice  experiments.  These  students  needed  little 
practice,  but  still  went  through  the  ten  practice  words. 
The  practice  words  were  used  as  the  teaching  device  for 
those  subjects  who  had  not  used  the  eguipment  before.  The 
need  to  speak  a  phrase  as  a  continuous  flow  of  speech  was 
explained,  as  was  the  explanation  of  the  mpaning  of  a 
guestion  mark  (?)  when  it  appeared  on  the  display.  If  the 
subject  showed  an  interest  in  the  machine's  capability 
during  the  the  practice  session,  their  guestions  were 
postponed  until  after  they  had  answered  fhe  guestionnaire. 

The  subjects  were  asked  to  ignore  all  the  errors  (if  any) 
which  occured  during  their  practice  session.  It  was 
explained  that  some  artificial   intelligence  algorithms  were 
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being  employed  and  the  system  was  select  inq  characteristics 
of  their  voice  for  use  in  the  main  portion  of  the 
experiment. 

As  the  subjects  were  going  through  the  200  word  list,  the 
experiment    was      stopped   each      time   an      error    occured.  The 

errcr  was  pointed  out  to  the  subject  and  a  short  explanation 
was  given  as  to  what  that  error  would  do  if  the  true  tex+ 
editor  was  employed.  The  idea  for  stopping  +he  program  was 
two  fold.  First  it  was  noticed  during  preliminary  program 
testing  that  some  people  started  to  read  the  words  on  the 
display  and  weren't  watching  the  recognized  word  displayed. 
In  ether  words,  the  experiment  bored  +  hem  and  they  weren't 
always  aware  of  the  errors.  At  first  this  had  been  solved 
by  placing  the  recognized  word  directly  under  ^he  prompt 
word.  Unfor+ unately  this  solution  caused  another  problem. 
The  subjects  who  participated  in  the  preliminary  testing 
started  to  say  the  recognized  word  instead  of  the  following 
prompt  word  which  had  a  devastating  effect  on  the  user's 
confidence  that  the  system  was  recognizing  +heir  voice. 
Therefore,  the  recognized  word  was  placed  to  the  right  of 
the  prompt  word,  and  each  error  was  pointed  out  to  the 
subject.  The  second  reason  for  stopping  the  program  was  to 
delay  the  subject  in  completing  the  experiment.  It  was 
hoped  that  the  idea  of  an  error  slowing  them  down  would 
transfer  to  their  perception  of  how  the  system  would  work  in 
a  real  environment.  The  plan  was  to  slow  them  down  thus 
creating   a    frustration    level    which    was   to    be    measured. 

There  were  numerous  times  even  with  the  precautions  taken 
where  the  subject  read  the  wrong  word,  or  started  to  make  a 
comment  without  the  microphone  being  turned  off.  This  led 
to  a  recognition  which  in  the  majority  of  cases  was  correct, 
when     the    spoken      utterance   was      obviously    incorrect.  The 

experimenter's    finger   was      always   kept   lightly   on      the    Apple 
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computer's  space  bar.  By  depressing  the  bar  the  microphone 
and  voice  recognition  system  were  deactivated.  The  majority 
of  the  time,  the  experimenter  depressed  the  space  bar  soon 
enough  to  avoid  a  correct  recognition  of  an  incorrect  voice 
input.  This  was  the  major  reason  for  the  delay  loop 
explained  earlier  in  the  program  software  description.  If 
the  space  bar  was  not  depressed  soon  enough,  and  the  correct 
response  appeared  on  the  display,  the  subject  was  told  that 
the  software  program  had  been  developed  to  do  its  own  data 
collection  automatically.  Furthermore,  it  was  explained 
that  the  experiment  was  interested  in  only  recognizer  errors 
not  the  human  errors  which  will  always  occur  with  a  voice 
system.  Therefore,  it  was  explained,  that  pressing  '-.he 
space  bar  was  an  automatic  override,  and  no  matter  what  was 
said  the  automatic  data  collection  routine  would  count  it  as 
a  correct  recognition.  This  explanation  seemed  to  satisfy 
everyone,  who  encountered  the  situation. 

After  the  subject  had  completed  the  200  words,  he  or  she 
was  asked  to  fill  out  a  two  page  questionnaire.  This 
questionnaire  is  attached  as  Appendix  D.  Questions  3,4  and 
5  created  the  data  which  was  of  most  importance  to  the 
experiment  and  was  the  measurement  of  user  acceptability. 
The  sets  of  response  alternatives,  for  these  three 
questions,  were  taken  from  an  Army  Research  Institute 
publication  on  questionnaire  construction  (1976).  The 
responses  have  been  tested  and  shown  to  have  mean  scaling 
factors  at  least  one  standard  deviation  away  from  each  other 
while  maintaining  the  parallel  wording.  Question  9, 
concerning  the  part  of  the  country  the  subject  grew  up  in, 
had  nothing  to  do  with  the  experiment  that  was  being 
conducted,  but  was  included  to  make  the  experiment  about 
testing  user  independent  algorithms  more  believable  to  the 
participants.   The  remaining  questions  are  self  explanatory, 
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and  will  be  reviewed  in  detail  when  discussing  the  analysis 
of    the    data    collected. 

After  the  questionnaire  was  completed  the  subject  was 
free  to  ask  questions  about  the  system.  They  were  led  to 
believe  that  all  algorithms  had  been  created  at  the  School, 
and  that  the  technology  was  not  commercially  ready  because 
of  the  extremely  large  amount  of  core  needed  to  run  the 
system.  This  was  done  to  insure  that  the  believability  in 
the  system  would  not  decrease  as  students  talked  about  the 
experiment.  On  ■'■he  other  hand,  it  was  not  the  in+ent  of  the 
experiment  to  lead  the  subjects  to  believe  that  voice 
recognition  capabilities  were  beyond  the  present  state  of 
the    art. 

The  experiment  was  a  total  success  as  far  as  the 
believability  of  the  system  was  concerned.  There  w=re  a 
couple  of  instances  when  the  random  number  generator 
cooperated  fully.  For  example,  a  subject  asked  a  question 
which  triggered  a  response  and  the  random  number  generator 
created      a    nonrecognition .  In      another      case   a      word      was 

misrecogn ized  in  the  test  sequence  and  the  same  word  was 
misrecognized  during  the  experiment,  both  times  the 
misrecognition      was      the      sound      alike      word.  There      were 

subjects  who  tried  to  analyze  the  system  and  hypothesized 
why      the    system      did      not      recognize   them      correctly.  One 

subject  was  convinced  that  any  word  with  an  "S"  sound  would 
not  be  recognized  properly  because  he  tended  to  slur  his  "S" 
sound.  There  were  only  two  subjects  oat  of  the  78  tested 
who  mentioned  the  fact  that  they  doubted  the  system  was 
recognizing     their   voice.  This  fact      was      noted   on      their 

guestionnaire   after    they    left   the   laboratory   area. 
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IV.  DATA  ANALYSIS 

Appendix  E  and  F  present  the  data  collected.  Appendix  E 
contains  the  raw  data  collected,  while  Appendix  F  has  the 
data  in  the  ranked  form.  All  of  the  data  analysis  used 
nonparametric  statistics  methodologies  based  on  ranks. 

The  first  column  of  data  in  Appendix  E  contains  the  total 
number  of  errors  the  user  observed.  This  number  is  the  sum 
of  nonr ecognit ions  and  both  types  of  misrecognitions.  The 
second  column  contains  the  tDtal  number  of  misrecognitions 
which  is  the  sum  of  the  sound  alike  errors  and  nonsense 
errors  which  were  observed  by  the  subject.  Columns  3,4,  and 
5  are  the  individual  error  totals  for  nonrecogr.i-'-ions,  sound 
alike  misrecognnitions,  and  random  misrecognitions, 
respectively.  Column  6  is  the  age  of  the  subject,  and 
column  7  is  the  average  number  of  hours  the  subject  spends 
using  a  computer  terminal  each  day.  The  average  number  of 
hours  spent  at  a  terminal  were  considered  important  for  the 
stock  brokers  and  airline  rese rvat ionists,  but  had  little 
meaning  for  the  students  who  became  the  only  participants  in 
the  experiment.  Therefore,  this  data  will  nof  be  used  in 
the  analysis,  but  is  presented  for  completeness. 

The  next  two  columns,  column  8  and  9,  are  the  answers  to 
the  guestions  on  whether  the  subject  had  ever  seen  or  used 
voice  recognition  equipment.  These  answers  are  coded  with  a 
1  representing  an  answer  of  "yes",  and  a  2  representing  a 
"no"  response.  Columns  10,  11,  and  12  are  the  responses  to 
the  guestions  dealing  with  user  acceptability.  These  are 
also  coded,  from  a  1  meaning  a  poor  acceptability  response 
to  5  for  a  high  acceptability  response.  Column  1 3  is  the 
sum  of  the  responses  to  the  three  questions.  Since  the 
guestions  each  had   parallel  wording  the  sum   of  the  answers 
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for  the  individual  questions  was  used  for  the  data  analysis. 
This  method  gave  a  more  accurate  numeration  of  the 
acceptability    level    for   each    subject. 

Column  14  and  15  are  the  data  collected  on  the  subject's 
perception  of  his  or  her  own  typing  ability  in  terms  of 
speed  and  accuracy  respectively.  rhess  data  fields  are  also 
numeric  codes  for  the  response  given  on  the  questionnaire. 
A  value  of  1  represents  a  poor  rating,  a  2  an  average  rating 
while  a  3  represents  a  very  good  rating  in  the  speed  and 
accuracy      capability   of      each   subject.  Column      16    is      the 

response  to  the  question  of  whether  the  subject  ever  had  a 
typing  course.  The  same  response  convention  was  used  for 
the  two    previous   yes-no   questions. 

The  second  to  the  last  column,  column  17,  tabulates  the 
data  collected  representing  the  geographic  region  where  the 
subject    grew   up.      The    codes   have   the    following    meaning. 

1  -  South 

2  -  East 

3  -  Midwest 
H   -  Foreign 

5  -  West 

6  -    All   over   or   not   specified 

Finally  the  last  column,  column  18,  is  the  subject's 
number  of  years  experience  with  computers  and  computer 
termina Is. 

As  it  was  already  mentioned  Appendix  F  contains  the 
ranked  data.  The  data  are  organized  in  xhe  same  manner  as 
previously  described  for  the  raw  data  except  some  of  the 
columns    are      the    ranks   of      the    data    collected.  There    were 

numerous  ties  and  the  rank  value  assigned  was  +he  average  of 
the  ranks  that  would  have  been  assigned  to  them  had  ■'-here 
been   no      ties.         All      of    the      recognition    error      counts    were 
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ranked,  as  were  the  age  data,  total  acceptability  rating 
data,   and   the   subject    experience   data. 

The  first  set  of  analysis  used  Spearman's  rank 
correlation  coefficient  (Conover,  1980)  as  a  test  statistic 
to  determine  if  there  was  any  negative  correlation  between 
the  number  of  errors  presented  to  a  subject  and  that 
subject's  acceptability  rating.  It  had  been  hypothesized 
that  as  the  number  of  errors  increased  the  user 
acceptability        would      decrease.  Therefore,         the         null 

hypothesis  was  that  the  number  of  errors  and  user 
acceptability  are  mutually  independent  or  had  no 
correlation.  The  alternative  hypothesis  was  that  these  two 
variables        are      negatively        correlated.  The        following 

tabulates  Spearman's  correlation  coefficient  calculated  on 
the    ranked   data    between    these   two  variables. 

all   errors  -.250 

total    misr ecogni tions  -.249 

non    recognitions  -.250 

sound   alikes  -.215 

random   misrecogn itions        -.201 

At  a  significance  level  of  .02  5  the  null  hypothesis  can  be 
rejected    for   the    first    three    values.  It    appears   as    if    the 

desired  negative  correlation  is  present,  but  the  correlation 
is  very  slight  as  indicated  by  the  value  of  the  correlation 
coefficient.  The  hypothesis  of  mutual  independence  can  not 
be  rejected  between  the  sound  alike  errors  and  user 
acceptability,  and  the  random  errors  and  the  acceptability 
v  a  r  i  a  bl  e . 

Figure  1  is  a  graph  of  the  numeric  user  acceptability 
totals  versus  total  user  perceived  errors.  It  shows  that 
although  there  is  evidence  of  a  negative  correlation  there 
is    little    information    existing    as   to      where    or   at    what    error 
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level  the  acceptability  values  start  to  decrease.  In  fact 
the  real  problem  is  exemplified  by  observing  4-he  average 
response    values      for    various    groups      of    subjects.  Table    1 

contains,  for  various  groups  of  subjects,  the  averages  of 
the  total  values  for  the  three  acceptability  questions  along 
with     the        standard      deviation.  There      is        very      little 

difference  between  the  groups.  Even  the  group  that  observed 
more  than  a  15%  error  rate  still  rated  the  system  in  the 
"like  it"  and  "would  use  it"  category.  A  Kr uskal-Wallis  test 
was  done  to  determine  if  these  groups  of  subjects  had  the 
identical    mean    response   values.  This   hypothesis   could    not 

be  rejected  at  the  .05  or  .1  significance  level.  The  test 
statistic  value,  T,  was  calculated  at  6.78  and  the  chi- 
square  distribution  quantile  for  the  four  degrees  of  freedom 
at  the  .05  level  is  9. 488  and  at  the  .1  level  is  7.779. 
Therefore,  even  though  a  small  negative  correlation  is 
detected  in  the  data,  very  little  information  can  be  gained 
as  to  where  a  distinct  drop  occurs  in  user  acceptability 
values. 

Two  other  Spearman's  Rho  correlation  coefficient  tests 
were  done.  First  a  correlation  possibility  was  investigated 
between  the  ranked  values  of  age  and  the  ranked  values  of 
acceptability.  The  hypothesis  that  age  and  acceptability 
values  were  not  correlated  could  not  be  rejected. 
Spearman's  correlation  coefficient  was  calculated  at  .03. 
The  same  test  was  done  to  check  the  data  for  mutual 
independence  among  the  ranked  values  for  years  of  experience 
and  the  ranked  values  of  the  acceptability  totals.  Again 
the  independence  hypothesis  crould  not  be  rejected  with  a 
correlation    coefficient   of  -.02. 

In  addition  to  the  Krus kal-wallis  test  previously 
mentioned,  a  series  of  similar  tests  vers  completed  in  order 
to      determine    if      there      were   any      differences      in    the      mean 
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acceptability  responses  among  different  groups  of 
individuals.  Table  2  summarizes  these  tests.  None  of  the 
hypotheses  that  all  the  mean  acceptability  responses  were 
idenitical  could  be  rejected.  In  other  words  no  statisical 
differences  could  be  found  among  the  various  groups  tested. 

The  only  test  which  suggested  a  possible  difference  was 
the  test  between  the  groups  divided  by  geography.  This  was 
interesting  because  numerous  people  approached  the 
experiment  asking  for  example,  if  this  machine  understood 
"Louisi anian".  They  knew  they  had  a  distinct  southern 
accent,  and  if  it  recognized  them  they  were  very  surprised. 
This  could  account  for  the  relatively  high  Kruskal- Wallis 
statistic  even  though  the  groups  could  not  statistically  be 
shown  to  have  different  means. 

In  conclusion,  there  is  really  very  little  information 
present  in  the  data  collected.  For  this  reason  the 
experiment  was  cut  short  before  the  time  was  spent  at 
business  establishments.  It  is  hypothesized  that  there  are 
at  least  two  basic  flaws  in  the  experimental  design.  First 
the  experiment  was  working  with  an  advanced  technology. 
Some  of  the  people  who  were  tested  were  not  aware  that  voice 
recognition  existed.  In  fact,  numerous  people  answered  on 
the  guest ionnaire  that  they  had  seen  voice  recognition  to 
computers  on  "Star-Trek".  With  an  at^i^ude  like  that  any 
recognition  capability  was  impressive.  If  this  is  true  it 
would  be  expected  that  there  might  have  been  a  larger 
difference  between  the  group  acceptability  averages  between 
those  who  had  seen  voice  recognition  equipment  before  and 
those  who  had  not.  Since  this  did  not  occur  we  can  only 
assume  that  the  group  which  had  seen  voice  recognition 
before  knew  about  its  user  dependence  limitations  and  were 
egually  impressed  with  the  user  independence  capabilities 
being  demonstrated. 
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The  second  problem  area  involves  the  lack  of  a  task  which 
needed  to  be  accomplished  within  the  experimental  framework. 
Reading  a  list  of  words  and  pointing  out  the  errors  did  not 
create  the  frustrating  situations  which  are  going  to  exist 
when  you  encounter  a  recognition  error  while  trying  to 
acccmplish  a  task.  Although  it  was  hoped  that  stopping  the 
experiment  each  time  an  error  occurred  would  provide  this 
feeling,  it  did  not  totally  simulate  the  frustration 
associated  with  task  completion.  Furthermore,  subjects  who 
had  never  seen  or  used  voice  recognition  eguipment  had  a 
difficult  time  visualizing  how  the  eguipment  would  actually 
be  used.  Although  an  explanation  was  given  to  each  subject 
at  the  beginning  of  the  experiment,  it  appears  as  if  the 
concept    was   not      totally    understood    by    everyone.  This    was 

evidenced  by  some  of  the  guestions  asked  by  the  subjects 
after  the  experiment  was  finished.  If  there  had  been  a 
realistic    job   or   goal,    this    problem      could    be    alleviated. 
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Graph  of  ''.aw  Data 
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Error  Rate 
Groups 


0.0  -  1.5 
2.0  -  5.0 
5.5  -  10.0 
10.5  -  14.5 
15.0  -  21.0 


#  of  sub j  ects 
in  group 


A 


19 
13 

2  5 
16 

7 


Average 


4.40 
4.  41 
3.  96 
3.  71 
3.  90 


Standard 
Dev  iat  ion 


4.  46 
4.  49 
4.  08 

3.  93 

4.  02 


TABLE  1. 
Acceptability  Response  Summary 
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GROUP 


TEST  STATISTIC 


CHI-SQUARE 


OS 


Seen  Voice  equipment- 
not  seen  voice  equipment 


1  .  18 


Used  Voice  equipment - 
not  used  voice  equipment 


3.  65 


Slow,  intermediate,  fast 
speed  typi  s t 


2.  14 


Poor,  fair,  very  good 
typing  accuracy 


4.  3 


Typing  course- 
no  typing  course 


0.12 


Part  of  country  raised 
in 


10.15 


+- 


3.  84 


3.  84 


5.  99 


5.  99 


3.  84 


i 

i 

■  -  i 
f 


11  .  07 


TABLE  2. 
Results  of  Kruskal -Wall  is  Tests 
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V.  RECOMMENDATIONS  AND  CONCLUSIONS 

The  data  collected  in  this  experiment  do  not  help  answer 
the  question  of  how  good  a  voice  recognition  system  must  be 
to  maintain  user  acceptability.  The  following 
recommendations  are  provided  for  future  research  efforts  and 
experiments. 

1.  Introduce  the  subjects  to  voice  recognition 
equipment  before  the  start  of  the  experiment. 

2.  Demonstrate  the  equipment  within  en  appropriate 
work  environment.  In  other  words,  demonstrate  the 
use  of  the  equipment  to  accomplish  a  suitable  job 
within  their  individual  areas  of  expertise. 

3.  Have  the  subjects  train  the  equipment.  Don'4- 
present  a  system  which  is  obviously  years  ahead  of 
currently  available  technology. 

4.  Set  up  a  series  of  tasks  which  are  suitable  for 
voice  recognition  input  which  can  be  accomplished 
within  the  test  environment.  It  appears  that  it 
is  important  that  the  job  is  very  realistic  in 
nature. 

5.  Consider  the  following  method  to  vary  the 
recognition  accuracy  for  the  experiment.  Use  a 
recognition  system  which  has  the  capability  to 
easily  access  not  only  the  word  which  was 
recognized,  but  the  runner  up  word.  Then  by  the 
use  of  a  random  process  determine  whether  +he 
recognition  unit  should  output  the  first  or  runner 
up  word.  If  the  recognition  system  used  has  a 
fairly  good  accuracy  rate,  the  the  first  choice 
word  should  be  the   correct  word.    Therefore,   by 
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randomly  selecting  the  second  word  you  are 
randomly  selecting  errors  from  the  recognition 
unit.  It  is  possible  that  the  recognition  unit 
will  make  a  mistake  and  the  recognized  word  will 
be  incorrect.  If  the  randoir  draw  determines  a 
correct  response  is  to  be  given,  the  first  word 
will  be  output  to  the  system  but  in  this  case  it 
is  an  error.  This  error  was  not  expected; 
therefore,  although  the  actual  accuracy  rate  of 
the  system  will  net  be  under  t he  experimenters 
control  the  overall  error  rate  should  be  very 
close  to  the  percentage  of  times  the  runner  up 
word  is  chosen.  If  a  100%  accurate  system  was 
used  this  percentage  of  runner  up  choices  would  be 
egual  to  the  error  rate  observed.  Since  a  100% 
accurate  system  does  not  exist  it  appears  as  if 
close  will  have  to  do. 

The  Interstate  Electronics  Corporation  machine  is 
suitable  for  this  type  of  task.  The  only  problem  involved 
with  this  recommendation  is  that  it  will  be  impossible  to 
observe  error  rates  much  less  than  the  underlying  error  rate 
associated  with  the  equipment  chosen  for  use  in  •'■he 
experiment. 

Since  there  was  evidence  of  the  hypothesized  negative 
correlation,  it  is  possible  that  the  frustration  measurement 
will  fulfill  the  needs  of  follow  on  experiments.  This 
question  of  frustration  measurement  should  be  investigated 
further  before  undertaking  the  next  phase  of  experimentation 
to  answer  the  guestion  about  acceptable  accuracy  rates  for 
voice  recognition  equipment. 


26 


BIBLIOGRAPHY 


Conover,  W.  J.,  Practical  Nonparametr ic  Statistics,  John 
Wiley  §  Sons,  1980. 

Jay,  G.  T.,  An  Experiment  in  Voice  Data  Entry  for  Imagery 
Intelligence  Reporting,  Master's  Thesis,  Naval  Postgraduate 
School,  Monterey,  1981. 

Nystrom,  C.  0.,  Questionnaire  Construction  Manual,  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences, 
1976. 

Poock,  G.  K. ,  Experiments  With  Voice  Input  for  Command  and 
Control:  Using  Voice  Input  to  Operate  a  Distributed  Computer 
Network  ,  Naval  Postgraduate  School  Report  NPS55- 80- 0 1 6 ,  April 
1980. 

Poock,  G.  K.,  A  Longitudinal  Study  of  Computer  Voice  Recognition 
Performance  and  Vocabulary  Size,  Naval  Postgraduate  School 
Report  NPS55-81-013,  June  1981. 

Scott  Instrument  Inc.,  VET/2  Voice  Entry  Terminal  Operator's 
Manual,  Scott  Instrument,  Denton,  Texas,  1981. 


27 


APPENDIX  A 


+LIST    1,9000 


10 
20 
30 
_40 
50 

G0 


"i*  = 

DIM 

DIM 


" "  ;    REM      D$    CONTAINS    A    CNTRL^D 

":     REM       It    CONTAINS    A    CNTRL    I 
F$(3):F*<1)    =    "COMPUTER" :F*(2)     = 


"STOCK" :F*(3)    =    "ftlRLINE" 


W$(200,2)_ 

PRINT    "DO    YOU" WANT    TO:":     PRTRT 
E    A    FILE":     PRINT 

INPUT  ft 

GOTO  1000, 2000, 3000,  4000 
"CREATE" 
E000 


"1.    CREATE    FTNEW   FItEn":~PRI Nr  " 2; 
3.     PRINTOUT    A    FILE":     PRINT    "4.     STOP" 


TJPDflT 


70   ON  A 
1000  B*  = 
1010   GOSUB 


1020   PRINT  'D*Y"OPEN 
1030  I  =  i:J  =  1 

1040 GOSUB  5000 

1050   GOTO  50 
2000  B$  =  "UPDATE" 
2010   GOSUB  E000 


;F$CA)  ;",  V001":  PRINT  D*T"  CLOSE"" 


202"0   QNERR  "  GOTO  20E0 

2030   PRINT  D*?"OPEN  " ;F$(A) 

2040   PRINT  D$?"READ  ";F*(A) 


V001 


2050   FOR  I  =  1  TO  200: 
2060   PRINT  D$",  "CLOSE" 

2070 PRINT  "DO  YOU  WANT 

TO 


FOR~T  =    1    TO   2:     INPUT    W*(I,J):    NEXT^T:- NEXT-T 


TO: 


2.  ADD 
20S0  INPUT 
2090  ON  Aft 
2100   PRINT 

0  STOP 
2110   INPUT 


THE~LIST  OF  WORD! 


PRINT    " 1. 
3RTNT 


CHANGE    INDIVIDUAL 
"STOP" 


WORDS":  PRINT  - 


ftA 

GOTO  2100,2200,2300  

"WHAT  WORD  NUMBER  DO~YOU  WANT  TO  CHANGE":  PRTN' 

INPUT  ft  -1" 

iz  IF  I  <  0  THEN   GOTO  2170 


2120   PRINT  "DO 
UND  ALIKE" 
2130   INPUT  J 


YOU  "WANT  TO  CHANGE"  :"  PRINT  "1.  THE  WORD":  PRTN" 


'"YOU^WANT'T 


-IE"  "SO 


J) 


2140   PRINT  I, J 

2150   INPUT  W$<:i 

21E0   GOTO  2 100 

2170  I  =  201:  GOSUB  5070 

21S0   GOTO  2070 

2200 PRINT  "NHftT  WORD 

I:  J  =  1 


NUMBER  DO  YOU  WANT  TO  START  AT 


2210 
2220 
2230 

2300 
3000 
J5010 
3020 
3030 
J5040 
3050 
30E0 
3070 


INPU' 

GOSUB  5000 

GOTO  207(3 

GOTO  50 
B*  =  "PRINTOUT" 
GOSUB  S000 


ONERR 
PRINT 
PRINT 
FOR  I 
PRINT 
PRINT 


GOTO  30E0 
D*: "OPEN  " ;F$(ft) ; 
D*?"READ  ";F$(ft> 


V001 


FOR 


30S0 
3  08  5 


=  1  TO  "200": 
D*: "CLOSE" 
"DO  YOU  WANT" : 
1ST  AND  SOUND  ALIKES,r 
INPUT  J 
ON  J  GOTO  3090, 3090, 


PRINT 
PR  IN 

>400 


JUST  THE 
"STOP77— 


~W<. 1 ,  J  )  :  NEXT~Jr-  NEXT  "T ' 
WORD  LIST":  PRINT  "2.  WORD  L 
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3090 
3100 
3110 
3120 
3130 
3170 


PRINT    "DO    YOU   HAVE~OKr=TWRGE~tOfiDED   RND~WANT  R   HRRDCOPY? 

INPUT    Y* 

IF  Y$  =  "N"  THEN   GOTO  3170 

PRINT 'D*?"PR#  2" : 

PRINT  I*, "SON" 

ON  J  GOTO  3200, 3300,3400 


(Y   OR  N) 


J 


"3500 

3210 

_3300 

3310 

3400 

j4000 

5000 

5010 

J5020 

5030 

5040 

5050 

5060 

5065 

_5070_ 

5080 

5090 

5100 


~F0R~~T~= r-rTO  200 

PRINT  D$; "PR#  0" 

FOR  I  -  1  TO  _200 
"  PRINT  D*;'rPR#  0,r 

GOTO  50 

STOP  :  END 

IF  I  >  200  THEN 

PRINT  I,  J 

INPUT  U$(I,J) 

IF~W*<IiJ>  = 
J  =  J  +  1 

IF  J  =  2  THEN   GOTO 


PRINT 


T7W*e:if  J)  : 
GOTO  3070 
PRINT  I, W$CI, J  - 
GOTO~3T370 


"NEXT! 

1),W*(I.J>:  NEXT  I 


GOTO  50G5 


'1 


"END"  THEN   GOTO  5065 


5010_ 
GOTO  5000 


6000 


6010 


I  =  I  +  l:j  =  1 

GOSUB  5070:  RETURN 

PR I  NT  _D*;" OPEN  ";F$(R) ; ",  V001" 

PRINT  D*; "WRITE  M?F*<A) 

FOR  K  =  1  TO  I  -  l:  FOR  J  =  1  TO 

PRI NT__D* ;  " CLOSE":  RETURN 

PRINT  "WHAT  FILE  DO  YOU  WRNT  TO  ";B*:  PRINT  " 1 .  COMPUTER" :~ PRrNT 
STOCK":  PRINT  "3.  AIRLINE" 
INPUT  R:  RETURN 


PRINT  W*CK, J):  NEXT  J:  NEXT  K 


.j 
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Airline  Vocabulary 


1. 

Allentown 

2. 

Abilene 

3. 

Albuquerque 

4. 

East  Hampton 

5. 

Washington  Dulles 

6. 

Columbus  Ohio 

7. 

Akron 

8. 

Ohara  Field 

9. 

Cleveland 

10. 

Cedar  Rapids 

11. 

Corpus  Christi 

12. 

San  Juan 

13. 

Boston 

14. 

Rapid  City 

15. 

Daytona  Beach 

16. 

Goose  Bay 

17. 

Dayton 

18. 

Utica 

19. 

Denver 

20. 

Lewiston 

21. 

Detroit 

22. 

Montpel ier 

23. 

Charleston 

24. 

Monterey 

25. 

Franklin 

26. 

Bangor 

27. 

Fargo 

28. 

Manitowoc 

29. 

New  York 

30. 

San  Jose 

31. 

Fort  Collins 

32. 

Laramie 

33. 

Kona 

34. 

San  Diego 

35. 

Freeport 

36. 

Missoula 

37. 

Oklahoma  City 

38. 

Colorado  Springs 

39. 

Pittsburgh 

40. 

Duluth 

41. 

Lousville 

42. 

St  Petersburg 

43. 

Honolulu 

44. 

Kirksville 

45. 

Columbia 

46. 

Columbus  Georgia 

47. 

Roanoke 

48. 

Rockford 

49. 

Washington  National 

50. 

Jacksonvil le 

Morgantown 

Aberdeen 

Kodiak 

Birmingham 

Washington  National 

Columbus,  Georgia 

Charleston 

Omaha 

Lakeland 

Grand  Rapids 

Cincinnati 

South  Bend 

Boise 

Sioux  City 

Long  Beach 

Pine  Coy 

Galveston 

Sitka 

Beaver 

Allentown 

Duluth 

Eau  Claire 

Kinston 

San  Jose 

Richland 

Boston 

Elko 

Roanoke 

Norfol k 

San  Diego 

Ft  Smith 

Miami 

Helo 

San  Francisco 

Bridgeport 

Honolul u 

Jefferson  City 

Hot  Springs 

Plattsburgh 

Detroit 

Knoxville 

Rochford 

Missoula 

Louisvil le 

Atlanta 

Columbus,  Ohio 

Manitowoc 

Medford 

Washington  Dulles 

Jackson 
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51. 

Atlantic  City 

52. 

Chi  co 

53. 

Amaha 

54. 

Plattsburgh 

55. 

Eau  Claire 

56. 

Norfolk 

57. 

Cheyenne 

58. 

San  Jose 

59. 

Phell ipsburgh 

60. 

Helo 

61. 

Medford 

62. 

Beaver 

63. 

Lewis  town 

64. 

Sitka 

65. 

Galveston 

66. 

Grand  Rapids 

67. 

Dubuque 

68. 

Pine  Coy 

69. 

Richland 

70. 

Long  Beach 

71. 

South  Bend 

72. 

Birmingham 

73. 

Sioux  City 

74. 

Cincinnati 

75. 

Kenston 

76. 

Acapulco 

77. 

Nome 

78. 

Orlando 

79. 

Pompano  Beach 

80. 

Harrisburg 

81. 

Hot  Springs 

82. 

Pittsfield  , 

83. 

Kodiak 

84. 

Milwaukee 

85. 

Knoxvill 

86. 

Ft.  Smith 

87. 

Paris 

88. 

Morgantown 

89. 

Springfield 

90. 

Elko 

91. 

New  Bedford 

92. 

West  Palm  Beach 

93. 

Bridgeport 

94. 

Kokomo 

95. 

Aberdeen 

96. 

Jackson 

97. 

Menomimee 

98. 

Fort  Wayne 

99. 

Rome 

100. 

Toledo 

101. 

Rochester 

102. 

Kalamozoo 

103. 

Providence 

Oklahmoa  City 

Toledo 

OHara   Field 

Pittsburgh 

Montpel ier 

New  York 

San  Juan 

Monterey 

St.    Petersburg 

Kona 

New  Bedford 

Denver 

Lewis  ton 

Utica 

Dayton 

Cedar  Rapids 

Albuquerque 

Goose  Bay 

Cleveland 

Daytona  Beach 

Cheyenne 

East  Hampton 

Rapid  City 

Corpus  Christi 

Akron 

Chi  co 

Rome 

Kokoma 

West  Palm  Beach 

Paris 

Colorado  Springs 

Springfield 

Dubuque 

Menominee 

Kirksvil le 

Fort  Collins 

Harrisburg 

Lewistown 

Pittsfield 

Fargo 

Phi  11 ipsburg 

Pompana  Beach 

Freeport 

Orlando 

Abilene 

Jacksonville 

Milwaukee 

Fort  Lauderdale 

Nome 

Acopulco 

Worchester 

Waterloo 

Provincetown 
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104.  Morristown 

105.  Clarksburg 

106.  Tallahassee 

107.  San  Angelo 

108.  Martha  Vineyard 

109.  Gainesville 

110.  Sacremento 

111 .  Lake  Charles 

112.  Osage  Beach 

113.  Brookings 

114.  Pueblo 

115.  Antiguo 

116.  Knoxville 

117.  Larado 

118.  Ogdensburg 

119.  Sumter 

120.  Show  Low 

121.  Wichita  Falls 

122.  Worthington 

123.  Provincetown 

124.  Waterloo 

125.  Apple  Valley 

126.  Appleton 

127.  Modesto 

128.  Worchester 

129.  Huntsville 

130.  Waterville 

131 .  Baton  Rouge 

132.  Marquette 

133.  New  Orleans 

134.  Walla  Walla 

135.  Tupelo 

136.  Astoria 

137.  Catskills 

138.  Atlanta 

139.  Janesville 

140.  Durango 
141  ,  Newburg 

142.  Trenton 

143.  Billings 

144.  Fayetteville 

145.  Greenvill 

146.  Buffalo 

147.  Evansville 

148.  Phoenix 

149.  Greensboro 

150.  Grand  Junction 

151 .  Anchorage 

152.  Wichita  Falls 

153.  San  Francisco 

154.  Waterville 

155.  San  Juan 

156.  Princeton 


Worthington 

Marthas  Vineyard 

Apple  Valley 

Show  Low 

Ogdensburg 

Knoxville 

Larado 

Lake  Tahoe 

Anchorage 

New  Orleans 

San  Angelo 

Al toona 

Watervil le 

El  Dorado 

Lynchburg 

Rochester 

Tampico 

Wichita 

Riverton 

Providence 

Kalamazoo 

Tallahassee 

Trenton 

Durango 

Sumter 

Catskills 

Janesville 

Osage  Beach 

Phoenix 

Bill ings 

Antingua 

Modesto 

Augusta 

Greinville 

Bermuda 

Huntsville 

Buffalo 

Clarksburg 

Grand  Junction 

Brookings 

Gainsesville 

Evansville 

Greensboro 

Jacksonville 

Marquette 

Ontario 

Morristown 

Baton  Rouge 

Tw  i  n  Fa  1 1  s 

San  Juan 

Greenville 

San  Jose 

Pendleton 
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157.  Churchill  Falls 

158.  Hastings 

159.  Wichita 

160.  Fort  Lauderdale 

161.  Pueblo 

162.  Saint  Louis 

163.  El  Dorado 

164.  Grand  Cayman 

165.  Jefferson  City 

166.  Idaho  Falls 

167.  Pendleton 

168.  Greenville 

169.  Miami 

170.  Grand  Canyon 

1 71 .  St  Croix 

172.  Lakeland 

173.  Harrison 

174.  Boise 

175.  Twin  Falls 

176.  Riverton 

177.  Ontario 

178.  Lynchburg 

179.  Altoona 

180.  Agusta 

181.  Victoria 

182.  Sarosota 

183.  Bermuda 

184.  Poplar  Bluff 

185.  Tulsa 

186.  Jacksonville 

187.  Poza  Rica 

188.  Guadalahara 

189.  Pensacola 

190.  Orlando 

191.  Topeka 

192.  Wausau 

193.  Galesburg 

194.  Penta  Gorda 

195.  Lake  Tahoe 

196.  Danville 

197.  Mankato 

198.  Hattiesburg 

199.  Lampico 

200.  Clinton 


Idaho  Falls 

Harrison 

Wichita   Falls 

Fort  Wayne 

Sacramento 

St.  Croix 

Pueblo 

Grand  Canyon 

Atlantic  City 

Churchill  Falls 

Princeton 

Waterville 

Laramie 

Grand  Cayman 

Saint  Louis 

Frankl  in 

Hastings 

Bangor 

Wichita  Falls 

CI inton 

Orlando 

Galesburg 

Astoria 

Wausau 

Topeka 

Pensacola 

Columbia 

Council  Bluffs 

Guadalahara 

Danville 

Walla  Walla 

Sarasota 

Poza  Rica 

Mankato 

Tillsa 

Victoria 

Newburg 

Tupelo 

Lake  Charles 

Fayetteville 

Pueblo 

Clarksburg 

Punta  Gorda 

Appleton 
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Computer  vocabulary 


1 

Login 

2 

Alter 

3 

Backward 

4 

Command 

5 

Cursor  Column 

6 

Findup 

7 

Left 

8 

Nfind 

9 

Put-D 

10 

Query 

11 

Reset 

12 

Set  Autosave 

13 

Set   File  Mode 

14  Set  Line  Character  Off 

15  Set  Number 

16  Set  Reserved 

17  Set  Synonym 

18  Set  Verify 

19  Stack 

20  Delete  Line 

21  Assign 

22  Five 

23  Six 

24  Seven 

25  Eight 

26  Nine 

27  Ten 

28  Alpha 

29  Bravo 

30  Charlie 

31  Logoff 

32  Bottom 

33  Cmsg 

34  C-Repalce 

35  Find 

36  Join  Cursor 

37  Msg 

38  Next 

39  Purge 

40  Replace 

41  Set  Arbitrary  Character 

42  Set   Filler 

43  Set  Line  Character  On 

44  Set  Nulls 

45  Set  Record  Format 

46  Set  Stream 

47  Set  Variable  Blank 

48  Split 

49  Uppercase 

50  Duplicate  Line 

51  Access 

52  Filedef 

53  Global 

54  Listfile 

55  Start 


Logoff 

Add 

Bottom 

Compress 

Cursor  File 

Forward 

Load 

Nfind-up 

Parse 

Quit 

Restore 

Set  Case 

Set  File  Name 

Set  Logical  Record  Length 

Set  Pack 

Set  Scale 

Set  Tableine 

Set  Wrap 

Status 

Duplicate  Line 

Access 

Four 

Sixty 

Seventy 

Eighty 

Ninety 

Send 

Papa 

Romeo 

Wh  i  s  key 

Login 

Backward 

Command 

Cursor  Column 

Findup 

Join 

Macro 

Nfind 

Put-D 

Reset 

Set  Autosave 

Set  File  Mode 

Set  Line  Character  Off 

Set  Number 

Set  Reserved 

Set  Synonym 

Set  Verify 

Stack 

Up 

Delete  Line 

Assign 

Fetch 

Cobol 

Filedef 

State 
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56 

Zero 

Kilo 

57 

One 

Echo 

58 

Two 

Echo 

59 

Three 

Thirty 

60 

Four 

Five 

61 

Profile 

No  Profile 

62 

Cancel 

C-Append 

63 

Cms 

Cmsg 

64 

Cp 

C-Replace 

65 

Expand 

Emsg 

66 

File 

Find 

67 

Join  Column 

Join  Cursor 

68 

Modify 

Move 

69 

Power- Input 

Preserve 

70 

Renumber 

Repeat 

71 

Selective  Ch 

ange 

Set  APL 

7? 

Set  Current 

Line 

Set  Escape 

73 

Set   Image 

Set   Implicityly  to   CMS 

74 

Set  Message 

Mode 

Set  Nondisplayable  Characte 

75 

Set  Prefix 

Set  Range 

76 

Set  Span 

Set  Stay 

77 

Set  Text 

Set  Truncate  Column 

78 

Sort 

Sos 

79 

Type 

Up 

80 

Debug 

Disk 

81 

Papa 

Alpha 

82 

Quebec 

Stack 

83 

Romeo 

Tango 

84 

Sierra 

Alpha 

85 

Tango 

Romeo 

86 

Uniform 

Nine 

87 

Victor 

Preserve 

88 

Wh  i  s  key 

X-ray 

89 

Yankee 

Char! ie 

90 

Zulu 

Move 

91 

No  Profile 

Profile 

92 

C-Append 

Cancel 

93 

C-Locate 

Cms 

94 

C-overlay 

Cp 

95 

Dupl  icate 

Down 

96 

Emsg 

Expand 

97 

Put 

Input 

98 

Join 

Join  Column 

99 

Move 

Msg 

100 

Preserve 

Purge 

101 

Repeat 

Replace 

102 

Set  APL 

Set  Arbitrary  Character 

103 

Set  Escape 

Seet   Filler 

104 

Set  Implicitly  Cms 

Set  Line  Character  On 

105 

Set  Nondispl. 

3yable  Characters 

Set  Nulls 

106 

Set  Range 

Set  Record   Format 

107 

Set  Stay 

Set  Stream 

108 

Set  Truncate 

Column 

Set  Variable  Blank 

109 

Sos 

Split 

110 

Up 

Uppercase 
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Ill 

Disk 

112 

X-Ray 

113 

Twenty 

114 

Thirty 

115 

Forty 

116 

Fifty 

117 

Sixty 

118 

Seventy 

119 

Eighty 

120 

Ninety 

121 

Xedit 

122 

C-Delete 

123 

C-Last 

124 

Compress 

125 

Cursor  File 

126 

Forward 

127 

Get 

128 

Load 

129 

Nfind-up 

130 

Overlay 

131 

Quit 

132 

Restore 

133 

Set  Case 

134 

Set   File  Name 

135 

Set  Logical   Word 

136 

Set  Pack 

137 

Set  Scale 

138 

Set  Tabline 

139 

Set  Wrap 

140 

Status 

141 

Top 

142 

Compare 

143 

Erase 

144 

Fetch 

145 

State 

146 

Delta 

147 

Echo 

148 

Foxtrot 

149 

Gulf 

150 

Hotel 

151 

Update 

152 

C- First 

153 

C-Insert 

154 

Copy 

155 

Cursor  Screen 

156 

Delete 

157 

Set 

158 

Help 

159 

Locate 

160 

On 

161 

Q-Quit 

162 

Read 

163 

Right 

164 

Set  Command  Line 

165 

Set   File  Type 

Length 


Debug 

Yankee 

Ninety 

Eighty 

Seventy 

Five 

Fi  f  ty 

Seven 

Thirty 

Twenty 

Update 

C- First 

C-Locate 

Copy 

Cursor  Screen 

Find-up 

Set 

Locate 

Next 

On 

Q-Quit 

Right 

Set  Command  Line 

Set  File  Type 

Set  Macro 

Set  Program  Function 

Set  Screen 

Set  Tabls 

Set  Zone 

Stack 

Transfer 

Cp 

Disk 

Filedef 

Start 

India 

One 

Findup 

Off 

Help 

Xedit 

Change 

C-Last 

Count 

Cursor  File 

Down 

Get 

Hextype 

Lowercase 

Off 

Query 

Recover 

Read 

Set  Column  Pointer 

Set  Hexidecimal 


Key 


if. 


166 

Set  Macro 

167 

Set  Program  Function 

168 

Set  Screen 

169 

Set  Tabls 

170 

Set  Zone 

171 

Transfer 

172 

CP 

173 

Cobol 

174 

India 

175 

Jul iett 

176 

Kilo 

177 

Lima 

178 

Mike 

179 

November 

180 

Oscar 

181 

Add 

182 

Change 

183 

Count 

184 

Down 

185 

Hextype 

186 

Input 

187 

Lowercase 

188 

Macro 

189 

Off 

190 

Parse 

191 

Recover 

192 

Save 

193 

Set  Column  Pointer 

194 

Set  Hexi decimal 

195 

Set  Mask 

196 

Set  Point 

197 

Set  Serial 

198 

Set  Terminal 

199 

Set  Equals 

200 

Shift 

Set  Mask 
Key  Set  Point 


Set  Serial 

Set  Terminal 

Set   Equals 

Type 

Compare 

Global 

Lima 

Delete 

Zero 

India 

Quit 

Oscar 

Alter 

Alter 

C-Insert 

C-Overlay 

Dupl icate 

Help 

Put 

Left 

Modi  fy 

Overlay 

Power-Input 

Renumber 

Selective  Change 

Set  Current  Line 

Set  Image 

Set  Message  Mode 

Set  Prefix 

Set  Span 

Set  Text 

Set  Prefix 

Sort 
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Stock  vocabulary 


1 
2 

3 
4 
5 
6 
7 
8 
9 
10 
11 
12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
23 
24 
25 
26 
27 
28 
29 
30 
31 
32 
3  3 
34 
35 
36 
37 
38 
39 
40 
41 
42 
43 
44 
45 
46 
47 
49 
50 
51 
52 
53 
54 
55 


Houston  Natural  Gas 

IBM 

ITT 

Iowa  Electric 

K-Mart 

Kaiser  Aluminum 

Lenox 

Leer-Siegler 

Lionel 

Iowa  Gas 

Litton 

Kaiser  Steel 

Lockheed 

MGM 

Magic   Chef 

Marriott 

Marathan  Oil 

Mattel 

Maytag 

Memorex 

McDonalds 

AMF 

McDonald-Douglas 

3M 

NCR 

Nabisco 

Natomas 

Oklahoma  Gas 

Penney1 s 

Combustion  Engineering 

Can  Edeson 

Dennys 

Macys 

Dow  Chemical 

Elgin 

Dorsey 

Florida  Power  &  Light 

Greyhound 

Hal iburton 

Kroehler 

Wisconsin  Power  &  Light 

Grumman 

U.S.  Air 

U.S.  Steel 

U.S.  Life 

Western  Union 

Zapata 

Air  Florida 

Wennebago 

Air  Wisconsin 

Atlantic  Richfield 

BDM 

Tampox 

Alabama  Power 


Oklahoma  Gas 

MGM 

IBM 

Iowa  Gas 

Gillette 

Kaiser  Steel 

Purex 

Singer 

Mattel 

Iowa  Electric 

Hal iburton 

Kaiser  Aluminum 

Square  D 

IBM 

Zenith 

Mattel 

Ashland  Oil 

Marriott 

Rubber  Maid 

Tampox 

McDonald-Douglas 

ATT 

McDonalds 

BDM 

CBS 

Sambos 

Ameroda  Hess 

Houston  Natural  Gas 

Macy's 

Cummins  Engine 

Western  Union 

Dorsey 

Penneys 

Dow  Jones 

Hal iburton 

Dennys 

Wisconsin  Power  &  Light 

Grumman 

Elgin 

Burroughs 

Florida  Power  &  Light 

Greyhound 

U.S.  Gypsum 

U.S.  Air 

U.S.  Steel 

Con  Edison 

Aenith 

Air  Wisconsin 

Whirl  pool 

Air  Florida 

Atlantic  Research 

MGM 

Zerox 

Florida   Power  &  Light 
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56 

57 

58 

59 

60 

61 

62 

63 

64 

65 

66 

67 

68 

69 

70 

71 

12 

73 

74 

75 

76 

77 

78 

79 

80 

81 

82 

83 

84 

85 

86 

87 

88 

89 

90 

91 

92 

93 

Q4 

% 

96 

97 

98 

99 

10.0 

101 

102 

103 

104 

105 

106 

107 

108 

109 

110 

111 


Bank  of  Virginia 

Burroughs 

CBS 

Atlantic  Research 

Zenith 

Xerox 

Bank  of  America 

John  Deer 

Delta  Airlines 

Dow  Jones 

Eastern  Airlines 

General  Motors 

General  Steel 

General  Radio 

Georgia  Pacific 

Goodyear 

Gillette 

Gulf  Oil 

Hicla  Mining 

Goodrich 

Heinz 

Pe  i  rex 

Sambos 

Singer 

Square  D 

Coll  ins  Radio 

Tandy 

Texaco 

United  Airl ines 

U.S.  Gypsum 

Ala  Moana 

Albertsons 

Gulf  Western 

Dome  Mining 

Ameroda  Hess 

American  Broadcasting 

Honda 

All  is  Chalmers 

American  Hospital 

Ashland  Oil 

Bank  of  America 

Bendvx 

Boise  Cascade 

Canadian  Pacific 

Ben  Gay 

Champion 

Chrysler 

Coca  Cola 

Clorex 

Chase  Manhattan 

Data  General 

Cities  Services 

Canoca 

Southern  Pacific 

Boeing 

Ford 


Bank  of  America 

Kroehler 

NCR 

Atlantic  Richfield 

Zapato 

Lenox 

Bank  of  Virginia 

Chrysler 

Eastern  Airlines 

Dow  Chemical 

Delta  Airlines 

General    Radio 

Data   General 

General   Motors 

Canadian   Pacific 

Goodrich 

K-Mart 

Gulf  Western 

Dome  Mining 

Goodyear 

Honda 

Clorox 

Collins  Radio 

Leer  Siegler 

Tandy 

Texaco 

Square  D 

Nabisco 

Eastern  Airlines 

U.S.   Life 

Coca   Cola 

Alles   Chalmers 

Gulf  Oil 

Hiclo  Mining 

Natomas 

American  Hospital 

Heinz 

Albertsons 

American  Broadcasting 

Marathan  Oil 

Bank  of  Cal  ifornia 

Ben  Gay 

Ben  Gay 

Georgia  Pacific 

Bendex 

Chose  Manhatten 

John  Deer 

Ala  Moana 

Memorex 

Champion 

General  Steel 

Citicorp 

Coca  Cola 

Georgia   Pacific 

Boise  Cascade 

Fotomat 
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112 

113 

114 

115 

116 

117 

118 

119 

120 

121 

122 

123 

124 

125 

126 

127 

128 

129 

130 

131 

132 

133 

134 

135 

136 

137 

138 

139 

140 

141 

142 

143 

144 

145 

146 

147 

148 

149 

150 

151 

152 

153 

154 

155 

156 

157 

158 

159 

160 

161 

162 

163 

164 

165 


El  Paso  Gas 

Union  Oil  of  California 

TRW 

Citicorp 

Revlon 

Woolworth 

Rockwell 

United  Technologies 

Fotomat 

San  Diego  Gas 

Southwestern  Gas 

Bell  &  Howell 

Honeywell 

Avon 

Threshold  Technology 

Allied  Corp. 

ACF  Industries 

Beckman  Instruments 

Big  Three 

Boston  Edison 

Brooklyn  Union  Gas 

Bulova 

Campbell  Soup 

Carolina  Power  &  Light 

Carpenter  Technology 

AMF 

Becton  Dickinson 

Carolina  Freight  Carriers 

Central  Maine  Power 

Clark  Oil  &  Refining 

Coldwell  Banker 

Colonial  Store 

Commonwealth  Edison 

Central  &  South  West 

Consolidated  Foods 

Continental  Can 

Colt  Industries 

Clark  Equipment 

Central  Soya 

Continental  Oil 

Copper  Range 

Consolidated  Freightway 

Continental  Air  Lines 

Corning 

Cummins  Engine 

Cooper  Laboratories 

Cyprus  Mines 

Del  Monte  Corp 

Continental  Telephone 

Detroit  Edison 

Diebold 

Dresser  Industries 

Eastman  Kodak 

Crocker 


San  Diego  Gas 

Bank  of  Cal  ifornia 

NCR 

Cities  Services 

Avon 

Whirlpool 

Honeywell 

Threshold  Technology 

Ford 

El  Paso  Gas 

San  Diego  Gas 

Rockwell 

Rockwell 

Revlon 

United  Technology 

Alles  Chalmers 

AMF 

Becton,  Dickinson 

Square  D 

Con  Edison 

San  Diego  Gas 

Burroughs 

Honeywell 

Wisconsin  Power  &  Light 

United  Technology 

ACF  Industries 

Beckman  Instruments 

Carolina  Power  &  Light 

Central  &  South  West 

Clark  Equipment 

Rockwell 

Colt  Industries 

Con  Edison 

Central  Soya 

Consolidated  Freightway 

Continental  Air  Lines 

Colonial  Stores 

Clark  Oil  &  Refining 

Central  &  South  West 

Continental  Telephone 

Cooper  Laboratories 

Consolidated  Foods 

Continental  Can 

Crocker 

Combustion  Engineering 

Cooper  Range 

Bucyrus-Erie 

Delta  Air  Lines 

Continental   Oil 

Commonwealth   Edison 

Di   Giorgio 

Dreyfess 

Eastern  Airlines 

Corning 
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166 
167 
168 
169 
170 
171 
172 
173 
174 
175 
176 
177 
178 
179 
180 
181 
182 
183 
184 
185 
186 
187 
188 
189 
190 
191 
192 
193 
194 
195 
196 
197 
198 
199 
200 


Bucyrus-Erie 

Emuson   Electric 

Empire  Gas 

Di    Georgia 

Dreyfus 

Faberge 

Federal   Mogul 

Fieldcrest 

Fairchild 

Emery  Air  Freight 

Ford  Motor 

General  Dynamics 

Gerber 

Grand  Union 

Hushey 

Hilton 

Host 

Inland  Steel 

Internation  Harvester 

Johnson  &  Johnson 

Kennecaft  Copper 

Lone  Star  Gas 

Lukens  Steel 

First  Chicago 

General  Food 

Holiday  Inn 

Foremost 

Hewlett  Packard 

Houston  Natural  Gas 

International  Paper 

Johnathan  Logan 

Magnavox 

Lone  Star  Industries 

Federal  Paper  Board 

Kimberly  Clark 


Cyprus  Mines 

Emery  Air  Freight 

Emery  Air  Freight 

Georgia  Pacific 

Dresser  Industries 

Fairchild 

Federal  Paper  Board 

First  Chicago 

Faberge 

Emerson  Electric 

Foremost 

General  Food 

Gillette 

Estern  Union 

Hewlett-Packard 

Holiday  Inn 

Houston  Natural  Gas 

Kaiser  Steel 

International  Paper 

Jonathan  Logan 

Kimberly  Clark 

Lone  Star  Industries 

Kaiser  Steel 

Fieldcrest 

General  Dynamics 

Hilton 

Ford  Motor 

Hershey 

Host 

International  Harverster 
Johnson  &  Johnson 
Lenox 

Lone  Star  Gas 
Federal  Mogul 
Kennecott  Copper 
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APPENDIX  C 


+LILIST    1,9000_ 


10      DIM    W$C200»2),  F*C3),  PC4) 

20J>*_-  " "  :  REM   CONTAINS  _A_C_ONTROL-D 

30  F$(l)    =    ,'CDMPUtER,,:F*(2)    =    "STOCK": F*C3)    »    "AIRLINE71 
40   PCI)    =    .99:P(2)    =    ,95:P<3)    =    .90:PC4)     =    .85 

45  M$  =  "     _^ __ 

"50   PRINT  "WHAT  WORD  DATABASE  DO  YOU  WARTTCrUSE?" ":'  PRINT  T.  XOMPUTER": 

"2.  STOCK":  PRINT  "3.  AIRLINE" 
E0   INPUT  ft  


70   PRINT  D$;"OPEN  " ; F*C A) ; " . V001 " 

80'  PRINT  D$?"READ  " ;F*CA) 

90   FOR  I  =  1  TO  200:  FOR  J  =  1  TO  2:  INPUT  W*CI,J):  NEXT  J:  NEXT  I 


100  PRINT  D$; "CLOSE  " 

105  PRINT  "INPUT  A  SEED":  INPUT  IS 

110  PRINT  "WHAT  ALGORITHM  DO  YQU  WANT  TO  USE?" 

T20  INPUT  B:  IF  B  <  0  THEN   STOP- 

130  GOSUB  40000 

140  CALL  JTABLE  +  15 


150  VOC$  =  "CLASSIC.  VOC" 

1G0   GOSUB  40100 

170   PRINT  "FIRST  SOME  PRACTICE  WORDS" 


1S0  I  =  1 

190   PRINT  I,W*CI,1):  GOSUB  40400:  FOR  Z  =  1  TO  150:  NEXT  Z:  GOGUB  300 

200   IF  I  <  11  THEN   GOTO  190 

210  Q  =  0:~S  =  0:N  =  0:T  =  "ET 

220   PRINT  "WE  WILL  NOW  START  THE  EXPERIMENT" 

230  1=1  •  

"240   PRINT  I,W*CI,1):  GOSUBT0400:  FDR  Z  =  1  TO  150:  NEXT  ZS  UUBUB  30GT 

250   IF  I  <  201  THEN   GOTO  240 

250   PRINT  "THANK  YQU  FOR  PARTICIPATING  IN  THE  EXPERIMENT" 

"270  T  =  (200  -  0  -  S  -  N)  /  .2 

2S0  PRINT  B;T; "-";Q? "-";S;"-" ;N 

290 GOTO  110 

300   IF   RND  CIS)  >  PCB)  THEN   GOTO  320 

310   PRINT  M$;W*(I»1):  GOTO  400 

320  R  =   RND  CIS) 

330   IF  R~<  .33  THEN   GOTO  390 

340   IF  R  <  . EB  THEN   GOTO  3E0 

350   PRINT  M$;"?" :Q  =  Q  +  l:  GOTO  400 


360  U  =   INT  C  RND  CIS)  *  200)  +  1 

370   IF  U  =  I  THEN   GOTO  360 

380   PRINT  M*?W*CU»1):N  =  N  +  l:  GOTO  400 


390   PR INT  M* ' W$ C 1 . 2 )  ' " S  =  S  +  1 

400      FOR    Z    =    1    TO    150:    NEXT    HI    =    I    +    l:     PRINT    :    RETURN 
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Appendix  D 


Code  # 


CATEGORY    [  ]  AIRLINE 

[  ]  STOCK  BROKER  Age  

[  ]  COMPUTER 

TIME  SPENT  AT  A  COMPUTER  TERMINAL  A  DAY  hrs  . 

□   YES 

□  NO 

2.  Have  you  seen  voice  recognition  equipment  used  before        r— r   YFS 

□  NO 

3.  Considering  that  you  know  how  many  typing  errors  you  normally  make,  and 
considering  the  number  of  errors  you  saw  the  voice  system  make,  how  well 
do  you  like  the  voice  system. 


Really 

Don't 

Neutral  : 

Like 

Really 

Dislike 

Like 

If  I   have  it,   fine 
If  I   don't  have   it, 

fine 

It 

Like   It 

□  □  □  □         □ 

4.  Is  the  accuracy  of  the  voice  recognition  system  adequate  enough  to  make  you 
want  to  use  it  in  your  daily  job? 

Very                        Slightly  Neutral  Slightly      Mery   Adequate 

Inadequate     Inadequate  Adequate 

□  □  □  □             □ 

5.  Comparing  voice  input  to  manual  typing  input,  is  voice  input: 

Undoubtedly    Moderately  The  Moderately    Undoubtedly 

Worse        Worse  Same  Better       Better 

□          □  a  □         □ 
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6.  Do  you  consider  yourself  a  slow,  intermediate,  or  fast  typist? 
Slow  Intermediate  Fast 

□  □  □ 

7.  How  accurately  do  you  think  you  type? 

Poor  Fair  Quite  Well 

□  □  □ 

8.  Did  you  ever  take  a  typing  course? 
Yes  No 

□  O 

9.  What  part  of  the  country,  (USA)  did  you  grow  up  in?    

10.  How  many  years  experience  have  you  had  in  typing  information  into  computers? 
years . 

11.  Please  tell  the  experimenter  how  much  and  what  type  of  educational  background 
you  have. 
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APPENDIX    E       Raw    Dat3 


C     1 


7    8    9    10    1  1     12     13    14     15     16    17    1 
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22 

20 

10 

12 

35 

1 

1 

1 

4 
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4 

13 

2 

2 

1 

1 

1 

38 

27 

1  1 

14 

13 

27 

2 

2 

2 

4 

4 

4 

12 

2 

2 

1 

2 

1 

36 

17 

19 

12 

c> 

32 

1 

2 

2 

3 

2 

2 

7 

2 

3 

2 

1 

14 

32 

21 

1  1 

10 

11 

27 

3 

2 

2 

4 

2 

5 

1  1 

1 

2 

1 

2 

1  1 

32 

21 

1  1 

15 

b 

28 

3 

2 

1 

4 

4 

4 

12 

1 

2 

1 

3 

1 

31 

20 

1  1 

9 

1  1 

38 

6 

4- 

2 

5 

5 

5 

15 

2 

2 

1 

1 

1 

30 

16 

14 

9 

7 

34 

1 

1 

2 

4 

H 

4 

12 

2 

2 

1 

1 

0 

29 

22 

7 

9 

13 

28 

2 

2 

1 

4 

5 

5 

14 

3 

3 

1 

1 

1  1 

29 

18 

1  1 

12 

6 

38 

2 

2 

2 

5 

4 

5 

14 

1 

3 

2 

4 

10 

29 

14 

15 

7 

7 

33 

1 

2 

1 

5 

5 

5 

15 

1 

1 

2 

3 

5 

29 

15 

14 

9 

6 

27 

2 

2 

1 

5 

4 

5 

14 

2 

3 

2 

1 

4 

29 

13 

1  1 

12 

6 

31 

5 

2 

1 

2 

1 

2 

5 

3 

3 

' 

2 

8 

28 

16 

12 

8 

8 

33 

2 

2 

o 

£ 

3 

5 

5 

13 

3 

3 

1 

3 

5 

28 

15 

13 

5 

10 

32 

1 

1 

1 

4 

4 

4 

12 

2 

2 

1 

3 

0 

26 

20 

6 

6 

14 

29 

2 

1 

1 

4 

4 

3 

11 

2 

2 

1 

3 

1 

24 

15 

9 

9 

6 

27 

2 

2 

2 

3 

2 

2 

7 

1 

3 

2 

3 

o 

24 

20 

4 

7 

13 

29 

2 

1 

1 

4 

2 

4 

10 

2 

2 

1 

2 

10 

23 

15 

8 

6 

0 

31 

2 

2 

2 

3 

4 

3 

10 

1 

2 

i 

2 

0 

23 

16 

7 

4 

12 

28 

5 

1 

1 

2 

2 

1 

5 

2 

3 

1 

o 

12 

22 

15 

7 

9 

b 

33 

1 

1 

1 

4 

5 

5 

14 

1 

1 

2 

2 

0 

22 

12 

10 

7 

5 
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12 
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3 
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21 
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3 
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5 

5 

5 

15 
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1 

2 

20 
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3 

0 

20 

13 

7 

& 
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38 

3 

2 

I 

2 

2 

4 

6 

2 

3 

1 

2 

14 

19 

U 

7 

7 

5 

29 

0 

2 

2 

4 

4 

4 

12 

2 

2 

1 

5 

0 

18 

12 

b 

5 

7 

26 

1 

2 

2 

5 

5 

5 

15 

2 

2 

1 

2 

3 

17 

11 

6 

5 
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4 
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2 
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3 

6 

17 

12 

5 

3 

9 

35 

3 

1 

1 

4 

5 

5 

14 

3 

3 

2 

3 

10 

17 

1> 

3 

5 

9 

30 

1 

2 

1 

4 

5 

4 

13 

2 

2 

1 

1 

2 

17 

:  i 

6 

5 

6 

27 
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2 

1 

5 

5 

5 

15 

2 

2 

2 

3 

1 

16 

8 

8 

6 

2 
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13 
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6 
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9 

3 
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34 
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1 
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6 
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12 
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6 
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